Inbound Integration RESTful Service

Overview

The RESTful API Interface Component streamlines energy market integration by providing a secure, standardised REST API for exchanging measurement data and market messages. This solution delivers value through:

Operational Excellence

• Automated message processing and validation
• Real-time error detection and reporting
• Reduced manual intervention
• Audit trails

Market Efficiency

• Faster processing of market messages
• Standardised data exchange
• Simplified market participant onboarding
• Flexible configuration options

Integration Capabilities

• Message submission and processing
• Status monitoring and health checks
• Validation rule management
• Audit logging and tracking
• Security token management

Component Technical Documentation

The IntegrationService REST interface is designed to handle various operations related to message processing. This documentation provides an extensive guide on how to use and test the REST interface, including the expected request format, testing tools, and a detailed flow of the program.

OpenAPI

type	url
Raw Spec	https://inrest.domain.com/docs/openapi.json
Scalar UI	https://inrest.domain.com/docs/scalar
Swagger UI	https://inrest.domain.com/docs/swagger

End-to-End Flow Description

Client Request : A client sends a message to the /v1/erex/aggregated_measurement_series endpoint.

Message Validation : The service validates the message using various validation services (e.g., GridRuValidationService, RuleBasedValidationService).

Error Handling : If validation fails, an error is logged, and an error response is sent back to the client.

Message Forwarding : If validation passes, the message is forwarded to the Energyworx file manager for further processing.

Response : A response indicating the result of the processing is sent back to the client.

Endpoints

Health Check

Endpoint : /healthz

Method : GET

Description : Returns a simple status message to indicate that the service is running.

Request : No request body required.

Response :

    {
      "status": "ok"
    }

Create Message

Endpoint : /v1/erex/aggregated_measurement_series

Method : POST

Description : Processes a message based on the provided request payload.

Request Headers :

Content-Type: application/json

Authorization: Bearer <SIGNED_JWT> : see security

Request Body : JSON object representing the message to be processed. The structure of the message is defined by the MessageDTO class.

    {
        "header": {
            "messageId": "<message_id>",
            "dateTime": "<date_time>"
        },
        "measurementSeries": {
            "reason": "<reason>",
            "erexReference": "<erex_reference>",
            "productId": "<product_id>",
            "energyUnit": "<energy_unit>",
            "ru": "<ru>",
            "periodStart": "<period_start>",
            "gridAreaSeries": [
                {
                    "gridArea": "<grid_area>",
                    "volumes": [
                        {
                            "position": "<position>",
                            "consumption": "<consumption>",
                            "generation": "<generation>"
                        }
                        // More Volume objects can be here
                    ]
                }
                // More GridAreaSeries objects can be here
            ],
            "referenceRevisedRequest": "<reference_revised_request>"
        }
    }

Replace the <...> placeholders with actual values. Note that reference_revised_request is optional and can be omitted. The grid_area_series and volumes are lists and can contain multiple GridAreaSeries and Volume objects respectively.

Response : JSON object representing the outcome of the message processing. The structure of the response is defined by the ResponseMessageDTO class.

      {
          "code": "<code>",
          "message": "<message>"
      }

Replace the <...> placeholders with actual values. Note that message is optional and can be omitted.

Security

The services require a signed jwt token. The signature is created with a serviceaccount key file (provided by energyworx). Thecreated JWT should have a payload similar to the following example:

    {
      "iss": SERVICE_ACCOUNT_EMAIL_ADDRESS,
      "sub": SERVICE_ACCOUNT_EMAIL_ADDRESS,
      "aud": TARGET_URL,
      "iat": IAT,
      "exp": EXP,
    }

• The iss and sub fields, specify the service account's email address. This is specified in the client_email field of the service account JSON file.
  Typical format: inrest_{NAMESPACE_ID}@{EWX_PROJECT_ID}.iam.gserviceaccount.com
• The aud field, specify the URL of the resource.
• The iat field, specifies the current Unix epoch time, and for the exp field, specify a time within 3600 seconds later. This defines when the JWT expires.

JWT signing python code example:

    import datetime
    import requests
    import jwt
    import json
    def generate_jwt_payload(service_account_email: str, resource_url: str) -> dict[str, str | datetime.datetime]:
        iat = datetime.datetime.now(tz=datetime.timezone.utc)
        exp = iat + datetime.timedelta(minutes=60)
        return {
            'iss': service_account_email,
            'sub': service_account_email,
            'aud': resource_url,
            'iat': iat,
            'exp': exp,
        }
    def sign_jwt_with_key_file(credential_key_file_path: str, resource_url: str) -> str:
        with open(credential_key_file_path, 'r') as credential_key_file:
            key_data = json.load(credential_key_file)
        print("Key Data:", key_data)  # Debug statement
        PRIVATE_KEY_ID_FROM_JSON = key_data.get("private_key_id")
        PRIVATE_KEY_FROM_JSON = key_data.get("private_key")
        SERVICE_ACCOUNT_EMAIL = key_data.get("client_email")
        if not PRIVATE_KEY_ID_FROM_JSON or not PRIVATE_KEY_FROM_JSON or not SERVICE_ACCOUNT_EMAIL:
            raise ValueError("Missing required key data in the credential file")
        additional_headers = {'kid': PRIVATE_KEY_ID_FROM_JSON}
        payload = generate_jwt_payload(service_account_email=SERVICE_ACCOUNT_EMAIL, resource_url=resource_url)
        signed_jwt = jwt.encode(
            payload,
            PRIVATE_KEY_FROM_JSON,
            headers=additional_headers,
            algorithm='RS256',
        )
        print("Signed JWT:", signed_jwt)  # Debug statement
        return signed_jwt
    CERT_PATH = "<path to your cert>"
    URL = "https://inrest.ewx-central-<tst|stg|prd>.ewxapis.com/v1/erex/aggregated_measurement_series"
    headers = {
        "Authorization": f'Bearer {sign_jwt_with_key_file(CERT_PATH, URL)}',
        "Content-Type": "application/json"
    }
    r = requests.post(URL, headers=headers, data=json.dumps({<dict witht he proper format>))
    print(r.text)

Flow of the Program

The Erex rest-service uses several services for different tasks such as validation, logging, and task creation. Key services include:

DatasourceConfigService

ValidationConfigService

GridRuValidationService

MessageIdValidation

LatestAndGreatestValidation

RuleBasedValidationService

AuditLoggerService

TaskService

Main Process

The main rest-service processes a Message object and returns a ResponseMessage object. It performs several validations on the message using the validation services. If any validation fails, it logs an error in the audit-logging-table, creates a processing task, and returns a ResponseMessage with the error code and message. If all validations pass, it forwards the message to the file manager of Energyworx with a payload type equal to the message type. The payload type can be used in Energyworx to set up automatic ingestion.

Important Rule-Based Validation

The RuleBasedValidationService validates a given message against a set of rules defined in a JSON file. It retrieves the validation rules for the given message type from the ValidationConfigService (GCP bucket rules_msgspec/<msg_type>.json). It is important the rules are present; otherwise, the program will fail.

JSON Structure for Validation Rules

    {
      "rules": [
        {
          "element": "NameOfElement",
          "validation": {
            "type": "range",
            "values": ["1", "2", "3"],
            "error_code": 1234,
            "error_msg": "range check fail"
          }
        }
      ]
    }

Each rule has an element specifying the name of the element to validate and a validation object specifying the type of validation, the values to use in the validation, and the error code and message to return if the validation fails.

GridRuValidationService

The GridRuValidationService extends MessageValidationService and validates the sequence of incoming grid and RU combinations in a message.

Validation Method

Extracts the RU and grid areas from the message.

Retrieves the EWX configurations using the DatasourceConfigService. This service first checks a Redis cache. If the configuration is not in the cache, it retrieves it from EWX.

Searches for datasource tags named source. These tags need a property active with a value of True to be picked up. The tag also needs the properties ru and grid_area, which are used in the validation.

After retrieving the config, it stores it in Redis with a TTL of 1 hour (changes in tags can take up to 1 hour to take effect).

The validate method checks if the RU and grid area combination from the message is in the EWX configurations. If it's not, it raises a SequenceValidationError with a specific error code and message. If the combination is in the EWX configurations, it returns a ValidationResult object with no error code or message.

In summary, the GridRuValidationService validates the sequence of incoming grid and grid-RU combinations in a message using the configurations retrieved from EWX.

Checking Files

Users can check files by going to the Energyworx console and searching the file manager for files with specific tags. They can also use advanced search and query integration-service-processing-logs.

Regression Testing

To perform regression tests:

Setup Test Environment : Ensure the test environment mirrors the production environment.

Test Scripts : Use automated test scripts to validate the functionality. Ensure all endpoints and validation services are tested.

Data Consistency : Verify that the data processed during the tests remains consistent and accurate.

Log Analysis : Analyze the logs in integration-service-processing-logs for any errors or discrepancies.

Comparison : Compare the results of the new implementation with the previous stable version to ensure consistency.